[Synthtrace] Adding Entities support #196258

cauemarcondes · 2024-10-15T09:02:53Z

known issue

- Transforms are not started by synthtrace. 
Because it duplicates data ingested by synthrace on signal indices. And it takes a long time to generate data.

- We are not able to open the Inventory page because of 👆🏻.

node scripts/synthtrace.js traces_logs_entities.ts --clean --live

or

node scripts/synthtrace.js traces_logs_entities.ts --clean --from=2024-04-08T08:00:00.000Z --to=2024-04-08T08:15:00.000Z

docs produces by the new scenario:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".entities.v1.latest.builtin_services_from_ecs_data",
        "_id": "2846700000000001",
        "_score": 1,
        "_source": {
          "service": {
            "name": "synth-node-trace-logs",
            "environment": "Synthtrace: traces_logs_entities"
          },
          "source_data_stream": {
            "type": [
              "traces",
              "logs"
            ]
          },
          "agent": {
            "name": [
              "nodejs"
            ]
          },
          "entity": {
            "id": "2846700000000001",
            "type": "service",
            "definitionId": "latest",
            "lastSeenTimestamp": "2024-10-15T08:56:20.562Z"
          },
          "event": {
            "ingested": "2024-10-15T08:56:20.562Z"
          }
        }
      },
      {
        "_index": ".entities.v1.latest.builtin_services_from_ecs_data",
        "_id": "2846700000000000",
        "_score": 1,
        "_source": {
          "service": {
            "name": "synth-java-trace",
            "environment": "Synthtrace: traces_logs_entities"
          },
          "source_data_stream": {
            "type": [
              "traces"
            ]
          },
          "agent": {
            "name": [
              "java"
            ]
          },
          "entity": {
            "id": "2846700000000000",
            "type": "service",
            "definitionId": "latest",
            "lastSeenTimestamp": "2024-10-15T08:56:20.562Z"
          },
          "event": {
            "ingested": "2024-10-15T08:56:20.562Z"
          }
        }
      },
      {
        "_index": ".entities.v1.latest.builtin_services_from_ecs_data",
        "_id": "2846700000000002",
        "_score": 1,
        "_source": {
          "service": {
            "name": "synth-go-logs",
            "environment": "Synthtrace: traces_logs_entities"
          },
          "source_data_stream": {
            "type": [
              "logs"
            ]
          },
          "agent": {
            "name": [
              "go"
            ]
          },
          "entity": {
            "id": "2846700000000002",
            "type": "service",
            "definitionId": "latest",
            "lastSeenTimestamp": "2024-10-15T08:56:20.562Z"
          },
          "event": {
            "ingested": "2024-10-15T08:56:20.562Z"
          }
        }
      }
    ]
  }
}

elasticmachine · 2024-10-15T09:03:00Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

achyutjhunjhunwala · 2024-10-15T09:18:54Z

packages/kbn-apm-synthtrace/src/lib/entities/entities_synthtrace_es_client.ts

+      if (timestamp) {
+        const isoString = new Date(timestamp).toISOString();
+        document['entity.lastSeenTimestamp'] = isoString;
+        document['event.ingested'] = isoString;


[nit]: Do you always want to override this field ? Or only when its not present ?

I always want to override it.

kpatticha · 2024-10-15T10:01:15Z

packages/kbn-apm-synthtrace/src/lib/apm/client/entities_synthtrace_kibana_client.ts

+    }
+  }
+
+  async uninstallEntityIndexPatterns() {


is it used anywhere?

No, but tests could use it after they run.

packages/kbn-apm-synthtrace/src/lib/apm/client/entities_synthtrace_kibana_client.ts

MiriamAparicio · 2024-10-15T11:26:54Z

packages/kbn-apm-synthtrace/src/cli/utils/start_live_data_upload.ts

    syntheticsEsClient,
    otelEsClient,
+    entitiesEsClient,
+    entitiesKibanaClient,


Why do we need the kibana client here?

https://github.com/cauemarcondes/kibana/blob/fe7ccdc7b060b6cb9b277ad32287980eb0b05cfb/packages/kbn-apm-synthtrace/src/scenarios/traces_logs_entities.ts#L44-L45

MiriamAparicio · 2024-10-15T11:28:18Z

packages/kbn-apm-synthtrace/src/cli/utils/synthtrace_worker.ts

      syntheticsEsClient,
      otelEsClient,
+      entitiesEsClient,
+      entitiesKibanaClient,


Same here, is kibana client needed for the entities?

https://github.com/cauemarcondes/kibana/blob/fe7ccdc7b060b6cb9b277ad32287980eb0b05cfb/packages/kbn-apm-synthtrace/src/scenarios/traces_logs_entities.ts#L44-L45

packages/kbn-apm-synthtrace/src/lib/shared/base_client.ts

MiriamAparicio

LGTM, just added some questions, for me to understand

elasticmachine · 2024-10-15T13:34:03Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: f9eae88
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-196258-f9eae88eb8f0

Failed CI Steps

Test Failures

[job] [logs] Jest Tests #5 / ColumnsPopover renders correctly a list of selected columns
[job] [logs] Jest Tests #13 / Response actions history page Read from URL params should read and expand actions using withOutputs
[job] [logs] Jest Tests #13 / Response actions history page Read from URL params should read and set command filter values from URL params

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/apm-synthtrace-client`	240	247	+7

Unknown metric groups

API count

id	before	after	diff
`@kbn/apm-synthtrace-client`	240	247	+7

History

💔 Build #242556 failed fe7ccdc
💔 Build #242474 failed d7c4e65

kibanamachine · 2024-10-15T15:21:52Z

Starting backport for target branches: 8.17, 8.x

https://github.com/elastic/kibana/actions/runs/11349217905

kibanamachine · 2024-10-15T15:26:42Z

💔 All backports failed

Status	Branch	Result
❌	8.17	The branch "8.17" does not exist
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 196258

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2024-10-16T15:50:01Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 196258 locally

cauemarcondes · 2024-10-17T09:07:00Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

## known issue ``` - Transforms are not started by synthtrace. Because it duplicates data ingested by synthrace on signal indices. And it takes a long time to generate data. - We are not able to open the Inventory page because of 👆🏻. ``` --- ``` node scripts/synthtrace.js traces_logs_entities.ts --clean --live ``` or ``` node scripts/synthtrace.js traces_logs_entities.ts --clean --from=2024-04-08T08:00:00.000Z --to=2024-04-08T08:15:00.000Z ``` docs produces by the new scenario: ``` { "took": 1, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 3, "relation": "eq" }, "max_score": 1, "hits": [ { "_index": ".entities.v1.latest.builtin_services_from_ecs_data", "_id": "2846700000000001", "_score": 1, "_source": { "service": { "name": "synth-node-trace-logs", "environment": "Synthtrace: traces_logs_entities" }, "source_data_stream": { "type": [ "traces", "logs" ] }, "agent": { "name": [ "nodejs" ] }, "entity": { "id": "2846700000000001", "type": "service", "definitionId": "latest", "lastSeenTimestamp": "2024-10-15T08:56:20.562Z" }, "event": { "ingested": "2024-10-15T08:56:20.562Z" } } }, { "_index": ".entities.v1.latest.builtin_services_from_ecs_data", "_id": "2846700000000000", "_score": 1, "_source": { "service": { "name": "synth-java-trace", "environment": "Synthtrace: traces_logs_entities" }, "source_data_stream": { "type": [ "traces" ] }, "agent": { "name": [ "java" ] }, "entity": { "id": "2846700000000000", "type": "service", "definitionId": "latest", "lastSeenTimestamp": "2024-10-15T08:56:20.562Z" }, "event": { "ingested": "2024-10-15T08:56:20.562Z" } } }, { "_index": ".entities.v1.latest.builtin_services_from_ecs_data", "_id": "2846700000000002", "_score": 1, "_source": { "service": { "name": "synth-go-logs", "environment": "Synthtrace: traces_logs_entities" }, "source_data_stream": { "type": [ "logs" ] }, "agent": { "name": [ "go" ] }, "entity": { "id": "2846700000000002", "type": "service", "definitionId": "latest", "lastSeenTimestamp": "2024-10-15T08:56:20.562Z" }, "event": { "ingested": "2024-10-15T08:56:20.562Z" } } } ] } } ``` (cherry picked from commit fe22ac9) # Conflicts: # packages/kbn-apm-synthtrace-client/index.ts # packages/kbn-apm-synthtrace/src/cli/scenario.ts # packages/kbn-apm-synthtrace/src/cli/utils/bootstrap.ts # packages/kbn-apm-synthtrace/src/cli/utils/start_live_data_upload.ts # packages/kbn-apm-synthtrace/src/cli/utils/synthtrace_worker.ts

# Backport This will backport the following commits from `main` to `8.x`: - [[Synthtrace] Adding Entities support (#196258)](#196258)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)

[Synthtrace] Adding Entities support

d7c4e65

cauemarcondes added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) v8.17.0 labels Oct 15, 2024

cauemarcondes requested review from a team as code owners October 15, 2024 09:02

botelastic bot added ci:project-deploy-observability Create an Observability project Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Oct 15, 2024

achyutjhunjhunwala reviewed Oct 15, 2024

View reviewed changes

kpatticha reviewed Oct 15, 2024

View reviewed changes

packages/kbn-apm-synthtrace/src/lib/apm/client/entities_synthtrace_kibana_client.ts Show resolved Hide resolved

comments

fe7ccdc

MiriamAparicio reviewed Oct 15, 2024

View reviewed changes

packages/kbn-apm-synthtrace/src/lib/shared/base_client.ts Show resolved Hide resolved

MiriamAparicio approved these changes Oct 15, 2024

View reviewed changes

removing old assets ref

f9eae88

achyutjhunjhunwala approved these changes Oct 15, 2024

View reviewed changes

cauemarcondes merged commit fe22ac9 into elastic:main Oct 15, 2024
30 checks passed

cauemarcondes deleted the apm-entities-synthtrace branch October 15, 2024 15:21

kibanamachine added the v9.0.0 label Oct 15, 2024

This was referenced Oct 16, 2024

[Inventory] List k8s entities in the grid #196155

Closed

[Inventory] Redirect k8s entities to their corresponding detail view #196142

Closed

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Oct 16, 2024

cauemarcondes mentioned this pull request Oct 17, 2024

[8.x] [Synthtrace] Adding Entities support (#196258) #196645

Merged

kibanamachine mentioned this pull request Oct 17, 2024

[Inventory] Adding initial e2e structure #196560

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Oct 17, 2024

kpatticha mentioned this pull request Nov 12, 2024

[Synthrace][EEM] Update data schema for EEM #183818

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Synthtrace] Adding Entities support #196258

[Synthtrace] Adding Entities support #196258

cauemarcondes commented Oct 15, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Oct 15, 2024

achyutjhunjhunwala Oct 15, 2024

cauemarcondes Oct 15, 2024

kpatticha Oct 15, 2024

cauemarcondes Oct 15, 2024

MiriamAparicio Oct 15, 2024

cauemarcondes Oct 15, 2024

MiriamAparicio Oct 15, 2024

cauemarcondes Oct 15, 2024

MiriamAparicio left a comment

elasticmachine commented Oct 15, 2024 •

edited

Loading

API count

kibanamachine commented Oct 15, 2024

kibanamachine commented Oct 15, 2024

kibanamachine commented Oct 16, 2024

cauemarcondes commented Oct 17, 2024

[Synthtrace] Adding Entities support #196258

[Synthtrace] Adding Entities support #196258

Conversation

cauemarcondes commented Oct 15, 2024 • edited by kibanamachine Loading

known issue

elasticmachine commented Oct 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MiriamAparicio left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 15, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

API count

History

kibanamachine commented Oct 15, 2024

kibanamachine commented Oct 15, 2024

💔 All backports failed

Manual backport

Questions ?

kibanamachine commented Oct 16, 2024

cauemarcondes commented Oct 17, 2024

💚 All backports created successfully

Questions ?

cauemarcondes commented Oct 15, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Oct 15, 2024 •

edited

Loading